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ABSTRACT 

Military  personnel  may  be  required  to  conduct  command  and  control  tasks  whilst  under 
various  levels  of  motion.  This  study  examined  the  usability  of  a  Battle  Management  System 
under  motion.  Usability  was  assessed  by  examining  performance  degradations  for  typical 
Battle  Management  System  tasks  that  a  commander  may  be  expected  to  complete  whilst  on 
the  move.  It  was  found  that  there  were  no  differences  in  task  performance  between  the  static 
and  mild  levels  of  motion,  but  there  were  differences  between  the  mild  and  high  levels  of 
motion.  It  was  also  found  that  participants  performed  better  at  tasks  in  which  they  were 
receiving  information,  rather  than  tasks  they  were  in  which  they  had  to  input  information. 
There  were  also  learning  effects  where  participants  performed  better  in  their  latter  experiment 
sessions. 
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Usability  of  a  Battle  Management  System  Under 
Simulated  Vehicular  Motion 


Executive  Summary 

Previous  research  has  found  that  motion  has  an  adverse  effect  on  people's  cognitive 
and  psychomotor  abilities.  This  has  implications  for  military  personnel  operating 
computer  based  equipment  in  military  vehicles.  This  report  covers  an  experiment  into 
the  usability  of  a  Battle  Management  System  (BMS)  under  different  levels  of  simulated 
vehicular  motion  in  order  to  perform  key  command  and  control  (C2)  tasks. 

The  experiment  simulated  six  different  types  of  BMS  tasks  including  reading  text, 
reading  an  enemy  unit's  location,  panning  and  zooming,  creating  text,  creating  a 
boundary  line,  and  creating  a  new  enemy  unit.  Participants  were  required  to  complete 
these  tasks  under  static,  mild  and  high  motion  conditions.  They  were  also  asked  to 
provide  a  subjective  rating  of  their  workload  and  the  difficulty  of  the  task.  A  Simulator 
Sickness  questionnaire  was  also  completed  by  participants. 

It  was  found  that  there  was  no  performance  difference  between  the  static  and  mild 
motion  conditions.  However,  there  was  a  performance  difference  between  the  mild 
and  high  motion  conditions.  This  was  replicated  in  the  subjective  responses.  No  effect 
of  simulator  sickness  was  observed. 

It  was  found  that  for  the  Pan  and  Zoom,  Read  Text,  and  Read  Unit  tasks,  participants 
performed  equally  amongst  the  three  motion  conditions.  However,  for  the  Create  Text, 
Create  Line,  and  Create  Unit  tasks,  performance  degraded  as  the  motion  increased. 

There  were  overall  learning  effects  across  the  six  experimental  sessions  and  for 
particular  tasks.  There  was  no  learning  effect  for  the  static  experimental  condition.  This 
indicates  that  participants  took  some  time  to  adjust  to  the  motion.  There  was  no 
learning  effect  for  the  Pan  and  Zoom,  Create  Unit,  Read  Text,  and  Read  Unit  tasks  in 
any  motion  condition  (i.e.  static,  mild  or  high).  However,  performance  did  improve 
over  sessions  for  the  Create  Text  and  the  Create  Line  tasks  in  the  high  motion 
condition.  Furthermore,  performance  improved  in  the  mild  motion  condition  for  the 
Create  Text  task. 

Key  conclusions: 

•  Under  high  motion  there  was  greater  difficulty  in  doing  typing  and  drawing 
tasks  than  the  Pan  and  Zoom,  Read  Unit,  Read  Text,  Create  Unit  tasks. 
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Subjects  required  a  number  of  sessions  to  adjust  to  the  motion.  Thus  training  in 
the  use  of  BMS  in  vehicles  should  include  a  motion  condition. 

This  experiment  has  identified  that  it  is  easier  to  read  rather  than  enter 
information  under  motion. 

Further  research  is  required  using  larger  motion  levels.  These  studies  will  be 
conducted  with  the  completion  of  DSTO's  Land  Motion  Platform  (LAMP). 
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1.  Introduction 


Cognitive  and  motor  functions  are  a  part  of  everyday  life,  and  something  which  are 
performed  countless  times  throughout  one's  life.  However,  performing  these  functions  in 
a  static  versus  moving  situation  is  different  and  in  the  military  context  this  difference  has 
operational  performance  implications.  This  report  outlines  research  into  how  motion 
affects  human  performance  while  carrying  out  standard  military  command  and  control 
tasks.  It  details  the  results  of  a  series  of  experiments  that  investigated  the  effect  of  varying 
levels  of  motion  on  the  typical  tasks  that  the  commander  of  a  military  vehicle  would  be 
expected  to  carry  out  using  a  command  and  control  computer.  Experiments  were  run 
using  a  three  degree  of  freedom  motion  simulator,  and  data  was  collected  and  verified 
using  a  wide  variety  of  technologies  such  as  inertial  measurement  units,  Matlab,  a  Battle 
Management  System  emulator  (BMSe).  Variables  of  interest  were  the  accuracy  of 
performing  the  tasks,  as  well  as  any  learning  effects  associated  with  the  tasks. 

1.1  Military  Context 

The  Australian  Army  is  about  to  receive  the  first  batch  of  a  new  digital  Command  and 
Control  (C2)  Battle  Management  Systems  (BMS)1.  As  part  of  this  roll-out,  the  system  will 
initially  be  installed  in  7  Brigade's  Bushmaster  vehicles  followed  by  the  7  RAR's  M113 
fleet.  Subsequent  phases  of  this  acquisition  are  likely  to  see  the  system  installed  into  the 
Army's  remaining  combat  vehicle's  (i.e.  ASLAVs  and  as  a  replacement  for  FBCB2  in  the 
M1A1  tanks)  as  well  as  in  current  and  future  logistics  vehicles  acquired  under  LAND  121. 
Figure  1  below  shows  the  system's  vehicle  mounted  computer  and  display. 


Brightness 

control 


Speaker 
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LCD  display 
with  Resistive 
Touchscreen 


Figure  1:  Neiv  BMS'  vehicle  mounted  computer  and  user  interface 


1  This  acquisition,  known  as  the  Battle  Group  and  Below  Command,  Control  and  Communications 
System,  or  BGC3  (but  also  known  as  BMS-C2)  is  a  joint  capability  acquisition  project  between 
LAND  75  (Combat  vehicles)  LAND  125  (Dismounted  Soldiers)  and  JP2072  (Land  tactical 
Communications  systems) 
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The  new  BMS  is  being  rolled-out  in  order  to  increase  Army's  tempo  of  operations  by 
decreasing  the  time  taken  to  share  key  C2  information  (i.e.  plans,  orders,  reports  and 
returns)  and  essential  situation  awareness  information  (friendly  and  enemy  force 
locations,  and  current  friendly  force  state).  The  extent  to  which  the  BMS  is  able  to  help 
achieve  this  goal  of  rapid  information  dissemination  however,  will  partially  depend  on 
how  easy  it  is  to  use  while  a  vehicle  is  on  the  move. 

1.2  Previous  Motion  Research 

There  has  been  a  great  deal  of  civilian  and  military  research  into  the  effect  of  motion  on 
human  performance.  This  research  has  generally  found  that  motion  tends  to  have  a 
detrimental  effect  on  task  performance  (Metcalfe  et  al  2008). 

This  research  has  also  shown  that  it  is  difficult  to  predict  the  strength  of  the  effect  of 
different  levels  of  motion  on  the  performance  of  specific  tasks  performed  in  particular 
contexts.  Seagull  and  Wickens  (2006)  for  example,  in  their  literature  review  of  the  effect  of 
vibration  on  task  performance  in  command  and  control  vehicles,  note  that  most  previous 
research  has  investigated  the  effect  of  motion  in  only  one  axis  (Vertical,  i.e.  up  -  down 
motion).  They  argue  however,  that  these  findings  cannot  (without  further  research)  be 
used  to  predict  the  effect  of  complex  multi-axis  random  motion  (as  experienced  in  military 
ground  vehicles)  on  task  performance.  Salmon  et  al  (2010)  in  their  review  of  previous 
research  into  the  effect  of  motion  on  the  use  of  in-vehicle  touch  screens  also  conclude  that 
the  effect  of  different  levels  of  motion,  on  different  in-vehicle  touch  screen  tasks,  remains 
ambiguous. 

This  suggests  that  accurate  measurement  of  the  effect  of  motion  on  BMS  task  performance 
must  use  realistic  tasks,  equipment,  and  motion  profiles.  The  best  way  to  do  this  is  to  use 
real  equipment  and  vehicles  in  real  operating  environments.  To  date  however,  there  have 
been  few  such  studies,  with  most  military  research  investigating  standard  usability  issues 
such  as  interface  design  by  using  static  BMS  systems  outside  of  vehicles  (i.e.  Command 
HQs  or  in  the  laboratory)  (for  example  see  Stanton  et  al  2010). 

Some  limited  research  on  vehicle  mounted  BMS  usability  in  the  field  has  been  conducted. 
Many  of  the  results  from  these  studies  however,  are  based  on  subjective  assessments  and 
not  objective  performance  data  (Seagull  et  al  2006).  This  is  because  it  is  difficult  to  control 
the  field  activity  so  that  motion  conditions  are  exactly  replicated  across  multiple 
experimental  runs  for  many  different  participants.  Those  studies  that  have  attempted  to 
collect  objective  data  (see  for  example  McDowell  et  al  2005,  and  Metcalfe  et  al  2008)  have 
generally  focussed  on  reach  time  and  accuracy  and  not  on  the  precise  effect  of  motion  on 
BMS  information  push  tasks,  such  as  text  entry,  drawing  and  icon  placement,  or 
information  pull  tasks,  such  as  reading  text  messages;  and  icons  and  drawings  on  digital 
maps  (Salmon  2010). 

One  way  to  gain  enough  control  to  accurately  collect  data  on  the  effect  of  motion  on  task 
performance,  is  to  use  motion  simulators  within  a  laboratory  environment2.  Once  again 
however,  most  of  the  studies  using  this  method  to  date  (see  Zywiol  et  al,  2006,  for 


2  These  simulators  can  be  made  to  partially  look  and  feel  like  real  vehicles  by  'mocking  up'  crew 
cabin  and  equipment  layouts. 
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example)  have  concentrated  on  identifying  reach  timing  and  accuracy  in  order  to  make 
recommendations  about  user  interface  design,  not  on  assessing  the  effect  of  motion  on 
BMS  task  performance  (and  therefore  on  operational  performance). 

Another  area  of  interest  is  determining  the  best  way  to  train  users  to  use  the  BMS  under 
motion.  There  is  some  evidence  to  suggest  that,  in  general,  tasks  are  best  learnt  in  realistic 
conditions  (Lenne  et  al  2010).  This  suggests  that  learning  BMS  tasks,  in  a  static  'classroom' 
environment,  may  be  less  than  optimal  for  use  under  motion.  Again  however,  there  is  a 
lack  of  research  into  the  effect  of  operator  experience  on  BMS  task  performance  under 
realistic  motion  conditions. 

Finally,  previous  research  has  identified  two  key  compounding  factors  that  make  it  more 
difficult  to  accurately  assess  the  effect  of  motion  on  task  performance.  The  first  factor  is 
whether  the  user  is  suffering  from  motion  sickness.  Motion  sickness,  depending  on  its 
severity,  also  has  an  effect  on  physical  and  cognitive  task  performance  under  motion  (Hill 
et  al  2005)  so  it  remains  difficult  to  separate  the  direct  effect  of  increases  in  motion  on  task 
performance  from  its  indirect  effect  via  induced  motion  sickness.  This  is  further 
complicated  by  the  fact  that  motion  sickness  tends  to  become  less  severe  under  higher 
workloads  and  after  longer  exposure  (Seagull  2006).  The  second  complicating  factor  is  the 
effect  of  individual  differences  on  task  performance.  In  the  case  of  performance  under 
motion  this  is  more  than  just  differences  between  individuals'  physical  and  cognitive 
abilities  (e.g.  typing  speed).  For  example,  individuals  also  vary  in  their  susceptibility  to 
motion  sickness  and  more  surprisingly  the  motion  frequencies  (resonances)  most  affecting 
individual  task  performance  under  motion  vary  according  to  anthropometric  factors  such 
as  individual  body  mass,  (Seagull  2006). 

1.3  DSTO  Motion  Research  Goals 

As  Army  is  acquiring  the  BMS  system  essentially  off  the  shelf,  the  most  pressing  research 
need  is  not  to  determine  the  best  user  interface  to  support  use  under  motion,  but  to 
determine  the  best  way  to  use  the  purchased  system  under  motion  (i.e.  identify  the  most 
effective  standard  operating  procedures).  DSTO's  research  program  in  this  area  therefore, 
aims  to  help  Army  maximise  the  overall  operational  impact  of  the  new  BMS  by  helping  to 
identify  the  'tipping  point'  when  the  loss  of  BMS  task  performance  (i.e.  time  and  accuracy) 
due  to  motion  begins  to  outweigh  the  potential  operational  benefits  provided  by  quicker 
information  dissemination  between  friendly  forces.  Therefore,  there  will  also  be 
implications  for  the  use  of  the  BMS  without  motion. 

Another  research  goal  is  to  determine  how  long  it  takes  to  learn  to  perform  a  particular 
BMS  task  under  motion  and  how  much  improvement  does  this  practice  bring?  This  has 
important  implications  for  determining  the  amount  of  'live'  BMS  practice  versus 
'classroom'  BMS  instruction  needed  to  maximise  performance.  It  is  also  hoped  that  the 
research  (described  below)  will  shed  light  on  the  extent  to  which  motion  simulation  can  be 
used  to  train  commanders  to  use  the  BMS  under  motion,  potentially  reducing  the  time 
needed  to  practice  in  the  field. 
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1.4  Previous  LOD  Motion  Research 

As  mentioned  above,  the  most  objective  and  controlled  way  to  determine  the  effects  of 
different  levels  of  motion  on  BMS  task  performance  is  to  use  laboratory  based  motion 
simulation  with  as  realistic  as  possible:  tasks,  equipment  and  environments3.  DSTO  has 
been  building  up  its  ground  vehicle  motion  simulation  capability  since  2009,  starting  with 
the  small  scale  motion  chair  described  in  Section  3.3.1  below  (also  see  Figure  2  below). 

1.4.1  Initial  DSTO  Pilot  Study 

The  motion  chair  was  first  used  during  a  DSTO  pilot  study  (Judd,  Demczuk,  and  Jacques 
2009)  which  looked  at  the  effect  of  motion  on  the  performance  of  six  key  BMS  tasks.  The 
goal  of  this  study  was  to  ensure  that  the  procedures  and  BMS  tasks  used  were  appropriate 
for  further  use  in  a  full  experiment  with  more  participants  (described  in  the  main  report 
below).  A  purpose  built  mock  up  of  a  real  BMS  was  used  with  the  motion  platform  chair,  a 
touch  enabled  17-inch  monitor  and  conventional  keyboard. 

Using  a  repeated  measures  design  with  motion  as  the  independent  variable,  three 
conditions  were  simulated:  no  motion,  mild  motion  (simulating  driving  on  a  sealed  road) 
and  high  motion  (simulating  driving  on  an  unsealed  road).  Participant  task  completion 
times  and  error  rates  were  then  measured  during  the  following  BMS  tasks: 

•  reading  text 

•  writing  text 

•  reading  a  unit's  location,  type  and  size,  on  a  digital  map 

•  panning  and  zooming  a  digital  map  using  digital  buttons 

•  creating  a  military  enemy  unit  on  a  digital  map 

•  drawing  a  multi-segment  control  line  on  a  digital  map. 

The  pilot  study's  results  suggested  that  motion  is  likely  to  affect  task  completion  times 
only  for  the  writing  text  task.  Task  error  rates  however,  were  higher  during  the  writing 
text,  creating  unit  and  drawing  line  tasks,  but  only  during  the  highest  motion  condition. 
Performance  also  improved  considerable  between  all  conditions  across  the  three  sessions 
of  the  experiment,  suggesting  that  even  more  improvement  might  be  possible  with  more 
practice. 

1.4.2  MU  ARC  Study 

The  pilot  study's  results  justified  further  experimentation  with  larger  numbers  of 
participants  in  more  realistic  simulated  environments.  DSTO  therefore  commissioned  the 
Monash  University  Accident  Research  Centre  (MUARC)  to  use  its  advanced  driving 
simulator  capability  to  perform  a  further  BMS  motion  task  performance  study  (see  Lenne 
et  al  2010  and  Salmon  et  al  2010).  MUARC's  simulator  consists  of  a  Holden  passenger 
vehicle  simulator  cab,  mounted  on  a  motion  platform  that  produces  realistic  road  feel  and 


3  DSTO  is  also  supporting  field  evaluations  of  the  BMS  under  motion  as  part  of  the  acceptance  and 
operational  test  and  evaluations  activities  supporting  the  initial  operating  capability  of  the  BMS. 
These  evaluations  however,  will  be  more  subjective  than  the  studies  described  in  this  report  and 
will  be  based  on  observations,  surveys  and  subject  mater  expertise. 
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passenger  vehicle  dynamics.  The  external  driving  environment  (of  a  rural  sealed  road  and 
country  side)  was  simulated  using  its  six  channel  projection  system  (one  displayed  at  the 
rear,  two  displayed  as  side  mirrors  and  three  at  the  front  displayed  on  a  curved  screen). 
The  same  BMS  emulator  and  set  of  six  BMS  tasks  used  in  the  DSTO  pilot  study  (described 
above)  was  displayed  on  a  ten  inch  LCD  screen  mounted  on  the  vehicle's  passenger-side 
dashboard.  The  20  participants  were  required  to  complete  the  six  tasks  (during  two 
repetitions  each)  while  experiencing  no-motion,  on-road  motion  or  off-road  motion.  The 
experiment  also  included  a  further  visual  task  requiring  the  20  participants  to  detect  and 
report  a  particular  type  of  object  in  the  surrounding  environment.  Additional  information 
about  subjective  workload  was  also  collected  using  surveys  but  also  by  measuring  the 
participant's  eye  fixation  and  blink  rates  (using  an  eye-tracking  camera)  as  well  as  by 
measuring  their  heart-rates. 

The  study,  which  has  since  been  published  (Salmon  et  al  2011)  found  that  the  effects  of 
high  levels  of  motion  on  the  speed  and  accuracy  of  information  'pull'  tasks  (e.g.  reading, 
icon  recognition)  are  marginal,  but  much  higher  on  information  'push'  tasks  (e.g.  text 
entry,  boundary  drawing).  These  findings  were  also  consistent  with  the  self  reported  and 
physiological  measures  of  workload  collected  during  the  experiment.  Task  completion 
times  however,  were  also  found  to  be  significantly  shorter  for  the  second  drive  in  each 
motion  condition,  indicating  a  potentially  significant  learning  effect,  suggesting  that  more 
training  under  motion  could  significantly  improve  performance.  Motion  however,  did  not 
effect  reaction  times  for  the  visual  detection  task,  yet  the  percentage  of  time  spent  gazing 
at  the  BMS  was  found  to  be  higher  in  the  high  motion  condition  (particularly  for  the  first 
drive).  During  the  experiment,  it  was  also  observed  that  reaching  strategies  changed  under 
motion,  with  many  participants  using  stabilising  points  for  their  hands  (e.g.  side  of  touch 
screen)4.  The  study's  authors  recommend  that  the  issues  relating  to  the  use  of  reaching 
strategies  be  further  examined.  They  also  recommend  that  the  results  should  be  cross¬ 
checked  by  further  studies  measuring  the  effect  of  motion  on  BMS  task  performance  in 
real  vehicles,  in  real  environments. 

1.5  Report  overview 

The  remainder  of  this  report  details  the  research  goals,  method  and  results  of  a  more 
complete  study  into  the  effects  of  motion  on  BMS  performance,  recently  completed  in 
Land  Operations  Division.  It  concludes  by  outlining  some  on-going  research  into  the  use 
of  the  BMS  under  motion  in  realistic  environments,  collected  during  recent  Land  121  Joint 
Protected  Mobility  Vehicle  -  Light  (PMVL)  trials  in  Puckapunyal.  The  report  will  finish  by 
discussing  some  of  the  proposed  future  studies  that  will  be  conducted  using  LOD's  LAnd 
Motion  Platform  (LAMP),  which  is  soon  to  be  commissioned. 


4  This  tendency  was  also  noted  during  the  initial  Pilot  study,  as  well  as  during  previous  studies 
(McDowell  et  al,  2005  and  Rider  et  al,  2003).  This  might  explain  why  the  writing  task  showed  the 
worst  performance,  as  stabilisation  points  for  this  task  were  hard  to  find  and  why  scores  on  pan 
and  zoom  tasks  were  nearly  100%  as  these  benefited  most  from  stabilising  on  the  side  of  the  touch 
screen. 
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2.  Rationale  of  Current  Study 

The  current  study  aimed  to  extend  the  DSTO  pilot  study  in  several  ways.  Firstly,  in  the 
pilot  study  it  was  found  that  participants  would  easily  learn  the  motion  file  sequence,  and 
they  would  often  wait  for  periods  in  the  file  with  low  motion  to  complete  the  tasks.  With 
the  current  study,  the  motion  profiles  were  more  random,  and  participants  could  not 
anticipate  periods  of  low  amplitude  motion  in  the  file.  The  motion  in  the  current  study 
was  also  standardised  according  to  British  Standard  BS  6841:1987,  titled,  "Guide  to 
Measurement  and  evaluation  of  human  exposure  to  whole-body  mechanical  vibration  and 
repeated  shock."  This  ensured  that  the  results  of  the  study  were  easily  comparable  to  real 
world  scenarios,  where  the  vibration  in  vehicles  can  be  measured. 

The  current  study  employed  the  use  of  seat  belts  which  the  pilot  study  did  not.  This 
ensured  that  all  participants  were  positioned  with  the  same  posture  and  had  the  same 
reach  to  the  computer.  This  avoided  the  situation  in  the  pilot  study,  where  it  was  observed 
that  participants  were  bracing  the  computer  in  different  ways  and  each  participant  had 
different  postures  when  seated. 

There  seemed  to  be  a  distinct  difference  between  the  information  that  was  being  "pushed" 
(i.e.  given)  to  a  commander  and  information  that  was  being  "pulled"  (i.e.  taken)  from  a 
commander.  The  pilot  study  showed  that  motion  affected  the  information  being  pulled 
from  the  commander  more  adversely  than  information  pushed  to  a  commander.  This 
study  looked  at  whether  degradation  could  eventually  be  found  for  the  "push"  tasks  with 
higher  amplitudes  of  motions.  It  also  looked  for  the  point  in  which  the  "pull"  tasks  became 
adversely  affected  by  motion  according  to  the  British  Standard. 

The  MU  ARC  study  and  the  DSTO  pilot  study  only  did  two  and  three  repeats  respectively 
of  the  BMS  tasks.  Both  previous  studies  had  found  some  learning  trends.  However,  neither 
had  the  repetitions  required  to  do  a  proper  learning  effect  analysis.  The  current  study 
employed  six  repetitions  of  the  BMS  tasks  as  well  as  training  of  the  BMS  tasks  before  the 
experiments  begun.  It  was  able  to  do  an  analysis  of  learning  effects  of  each  task  type  over 
the  various  motion  conditions. 

2.1  Aims 

There  were  several  aims  of  the  current  study.  Firstly,  there  is  a  need  to  determine  how 
different  levels  of  motion  affect  performance  on  C2  tasks.  Three  levels  of  motion  were 
chosen  for  the  study,  static,  mild  and  high.  The  mild  motion  was  a  "level  two"  (out  of  six) 
according  to  the  British  Standard,  and  the  high  motion  was  a  "level  four"  according  to  the 
standard.  The  aim  was  to  see  at  what  level  motion  would  have  a  significant  effect  on 
performance.  The  mild  motion  is  typical  of  that  experience  on  a  bitumen  road.  The  high 
level  is  typical  of  a  rough  dirt  road. 

Another  aim  of  this  study  was  to  determine  which  types  of  C2  tasks  are  most  adversely 
affected  by  motion.  From  the  pilot  studies,  it  would  appear  that  it  is  easier  for 
commanders  to  receive  information,  but  much  harder  for  them  to  give  information. 
However,  the  results  in  the  pilot  study  did  not  yield  any  significant  effects,  rather  just 
trends  were  observed.  Hence,  this  study  aimed  to  further  investigate  this  trend. 
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Lastly,  the  study  looked  at  whether  participants  could  'learn'  to  cope  with  the  motion  at 
the  various  levels.  If  performance  under  motion  can  be  improved  by  repeated  exposure, 
then  this  could  potentially  alter  commanders  training  regimes  and  plans.  Commanders 
could  be  better  prepared  to  perform  BMS  tasks  under  motion. 

Based  on  previous  literature,  and  past  studies  (the  DSTO  pilot  study,  and  MU  ARC  study), 
the  following  hypotheses  were  created: 

2.2  Hypotheses 

1.  There  will  be  differences  in  performance  for  the  high  versus  static  level  of  motion,  but 
not  for  the  mild  versus  static  motion. 

2.  Participants  will  perform  better  on  tasks  in  which  they  read  information  compared  to 
tasks  in  which  they  have  to  enter  information. 

3.  There  will  be  learning  effects  based  on  type  of  task,  and  motion. 


3.  Research  Method 

3.1  Participants 

There  were  16  (13  male,  three  female)  Defence  Science  and  Technology  Organisation 
(DSTO)  employees  who  participated  in  the  experiment.  The  average  age  of  the  participants 
was  32  years  (SD  =  8.02),  with  participant  age  ranging  from  22  to  47  years.  Nine  of  the 
participants  had  military  experience,  and  seven  participants  also  had  some  form  of 
experience  using  command  and  control  systems  in  the  past.  Two  of  the  participants  were 
left  handed,  and  one  participant  had  slight  colour  vision  impairment.  All  participants 
willingly  volunteered  to  take  part  in  the  study  without  any  form  of  compensation. 

3.2  Design 

The  study  looked  at  two  critical  aspects  of  command  and  control  tasks,  which  were,  task 
performance  and  learning  effects  of  doing  tasks  repeatedly  over  time.  It  employed  a  (3  x  6) 
repeated  measures  design  that  investigated  three  different  levels  of  motion  (static,  mild, 
and  high),  and  six  task  types  (pan  and  zoom,  reading  text,  reading  unit  icons,  creating  text, 
creating  boundary  lines,  creating  unit  icons). 

3.3  Materials 

3.3.1  DBox  motion  chair 

To  simulate  vehicle  motion  a  D-BOX  GP-PRO-200  'Gaming'  motion  platform,  shown  in 
Figure  2,  which  is  a  commercial  product  designed  to  replicate  racing  car  motions  while 
playing  computer  games  was  used.  The  D-BOX  is  capable  of  three  types  of  motion;  roll  of 
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approximately  four  degrees  (left  right  rotational  motion),  pitch  of  four  degrees  (forward- 
backward  rotational  motion;  and  heave  (vertical  linear  displacement  or  up  down  motion 
of  up  to  40mm).  The  chair  can  simulate  accelerations  of  up  tolg  (i.e.  9.8  metres  per  sec2). 

Apart  from  the  BMS  screen  there  were  no  simulated  external  environment  visuals.  This 
was  deliberately  done  to  minimise  simulator  sickness  due  to  visual  cues/ miscues. 


Figure  2:  DBox  Motion  Platform 
3.3.2  XSens  IMU 

An  XSens  MTx  inertial  measurement  unit  (IMU)  combined  with  the  XSens  Xbus  Master 
was  used  to  measure  the  vibrations  that  were  being  experienced  by  the  participants,  as 
shown  in  the  Figure  3.  Data  was  sampled  at  120Hz  at  a  baud  rate  of  921600.  Further  details 
on  the  sensor  can  be  found  at  http://www.xsens.com/en/industrial- 
applications/ mtx?tab=l. 


Figure  3:  XSens  motion  sensors 
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3.3.3  Panasonic  Toughbook 

The  command  and  control  tasks  were  installed  on  Panasonic  Toughbook  CF-19,  which  has 
a  10.4"  1024  x  768  VGA  touchscreen.  The  computer  was  attached  to  the  mount,  which  was 
available  in  front  of  the  participant,  as  can  be  seen  in  Figure  1.  For  more  information  see 
http:/ / www.buytough.com/ tb_19.asp. 

3.3.4  BMS  emulator 

The  Battle  Management  System  (BMS)  emulator  was  produced  to  create  a  comparable 
replica  of  a  real  Battle  Management  System.  It  is  capable  of  doing  similar  tasks  such  as 
reading,  writing,  pan  and  zoom,  creating  enemy  units,  creating  boundary  lines,  and 
detecting  enemy  units  on  a  fictitious  map.  A  screenshot  of  the  emulator  can  be  seen  below. 
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Figure  4.  Screenshot  of  the  BMS  task  emulator  user  interface. 

3.3.5  Subjective  Workload  Measures 

A  series  of  subjective  questionnaires  were  used  for  the  experiment.  These  included  a 
motion  sickness  susceptibility  questionnaire,  simulator  sickness  questionnaire,  the  NASA 
Task  Load  Index  questionnaire,  and  a  Task  Difficulty  questionnaire.  A  copy  of  the 
questionnaires  can  be  seen  in  Appendices  A,  B,  C,  and  D  respectively.  The  motion  sickness 
questionnaires  were  used  to  determine  whether  participants  were  susceptible  to  motion 
sickness  or  felt  motion  sick  after  simulation  exposure.  There  was  also  a  demographic 
questionnaire  given  before  the  start  of  the  experiment  which  can  be  found  in  Appendix  E. 
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3.3.6  Motion  Sickness  Questionnaires 

3.3.6. 1  Short  Motion  Sickness  Susceptibility  Questionnaire 

An  initial  assessment  of  the  participants'  susceptibility  to  motion  sickness  was  assessed  by 
the  Short  Motion  Sickness  Susceptibility  Questionnaire  (Appendix  A).  If  it  was  found  that 
participants  were  susceptible  to  motion  sickness,  it  would  be  recommended  that  they  do 
not  continue  with  the  study. 

3. 3. 6.2  Simulator  Sickness  Questionnaire 

A  questionnaire  following  motion  stimulus  was  given  to  determine  whether  participants 
became  ill  as  a  result  of  being  subjected  to  motion.  The  questionnaire  can  be  found  in 
Appendix  B. 

3. 3. 6. 3  NASA  Task  Load  Index 

After  completing  each  BMS  condition,  subjects  filled  in  two  workload  assessment 
questionnaires.  The  NASA  Task  Load  Index  (TLX)  was  used  to  distinguish  their  subjective 
workload  scores  (Hart  &  Staveland,  1988).  The  TLX  is  a  multi-dimensional  rating 
assessment  with  six  subscales:  Mental  Demands,  Physical  Demands,  Temporal  Demands, 
Own  Performance,  Effort,  and  Frustration.  Each  subscale  is  presented  as  a  line  divided 
into  20  equal  intervals  anchored  by  bipolar  descriptors  (e.g.  High/Low).  The  questionnaire 
was  presented  after  each  condition. 

The  TLX  was  in  the  form  of  the  Raw  TLX  (RTLX),  a  common  modification  to  the  original 
which  eliminates  the  weighting  process  of  the  sub-scales.  The  ratings  are  simply  averaged 
or  added  to  create  an  estimate  of  overall  workload.  This  method  has  been  considered  as 
equally  sensitive  to  the  original  (Byers,  Bittner  &  Hill,  1989).  During  the  training  session, 
discussion  of  the  TLX  was  used  to  help  subjects  calibrate  their  understanding  of  the  rating 
scales.  The  scores  were  either  added  or  multiplied  by  20  to  obtain  a  normalised  maximum 
score  of  100.  The  Performance  scale  was  excluded  from  Total  RTLX  %  results  as  it  was  felt 
this  was  more  indicative  of  mental  load. 

3. 3. 6.4  Task  Difficulty  Questionnaire 

A  Task  Difficulty  questionnaire  was  also  completed  by  subjects  after  each  BMS  condition. 
Subjects  scored  difficulty  for  each  of  the  six  BMS  tasks  after  each  of  the  three  motion 
conditions.  A  five  point  scale  was  used  with  subjects  circling  a  number  from  one  (not 
difficult)  to  five  (extremely  difficult).  The  scores  were  multiplied  by  20  to  obtain  a 
normalised  maximum  score  of  100. 

3.4  Procedure 

3.4.1  Creating  Motion  Profiles 

The  British  Standards  BS  6841:1987,  titled,  "Guide  to  Measurement  and  evaluation  of 
human  exposure  to  whole-body  mechanical  vibration  and  repeated  shock,"  was  used  to 
create  motion  profiles  which  simulated  two  particular  levels  of  the  Standards.  Three 
random  values  (roll,  pitch  and  heave)  were  inputted  into  the  DBox  using  an  executable 
file.  The  files  were  assessed  using  the  XSens  IMU  and  appropriate  scaling  was  applied  in 
order  to  gain  the  correct  frequency  magnitude  of  vibration  in  line  with  the  Standards. 
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Three  motion  conditions  were  used  for  the  experiment.  One  was  static,  the  other  was  mild 
and  lastly  a  high  motion  condition.  The  mild  motion  was  a  "level  two"  motion  according 
to  the  standards,  with  a  root  mean  square  (RMS)  value  of  0.47m/ s2.  The  high  motion  was  a 
"level  four"  motion,  with  an  RMS  value  of  1.2m/  s2. 

3.4.2  Participant  Instructions 

Participants  were  given  instructions  on  how  to  complete  the  BMS  tasks  and  they 
completed  two  training  sessions  to  ensure  that  they  fully  understood  all  the  instructions. 
They  were  informed  of  the  voluntary  nature  of  the  experiment  and  advised  to  indicate 
whether  they  felt  motion  sick  at  any  point.  Participants  were  also  shown  the  various 
questionnaires  that  they  would  be  required  to  fill  out  throughout  the  experimental 
sessions.  There  were  slightly  different  instructions  for  the  first  session  compared  to  the 
subsequent  experimental  sessions. 

At  the  start  of  each  experimental  session,  participants  were  instructed  to  complete  the  SSQ, 
and  then  proceed  to  complete  the  first  component  of  the  experiment,  which  was  either  a 
static,  mild  or  high  level  motion  condition.  The  condition  was  completed,  participants 
were  asked  to  complete  the  NASA  TLX  and  task  ranking  questionnaire.  Furthermore,  if 
participants  just  completed  a  motion  condition,  they  were  also  asked  to  complete  the  SSQ. 
This  procedure  was  repeated  until  all  three  condition  for  the  session  were  completed.  The 
complete  script  given  to  all  participants  can  be  found  in  Appendix  F.  Participants  also 
observed  the  experimenter  demonstrate  how  to  do  the  tasks. 

3.4.3  Scheduling  and  Capturing  Data 

Participants  were  scheduled  to  do  one  session  per  week.  However,  due  to  the  participants' 
other  work  commitments,  this  was  not  always  the  case.  At  the  beginning  of  each  session  a 
static  training  condition  was  administered.  Following  the  training,  each  session  consisted 
of  a  static,  mild  and  high  motion  condition  presented  in  random  order.  During  each 
condition  they  completed  the  BMS  tasks.  Each  set  of  the  BMS  tasks  consisted  of  24  items 
(four  of  each  type:  Pan  and  Zoom,  Read  Text,  Read  Unit,  Create  Text,  Create  Line,  and 
Create  Unit)  presented  in  a  random  order.  The  training  condition  was  provided  to  ensure 
no  participants  forgot  how  to  complete  a  particular  task. 

A  program  was  written  to  capture  the  data  that  was  inputted  by  the  participants.  Each 
response  given  by  the  participant  was  recorded  automatically  in  a  '.csv'  file  format. 

3.4.4  Task  Types 

There  were  six  different  tasks  that  the  participants  had  to  complete.  Participants  were 
asked  to  complete  all  tasks  as  quickly  and  accurately  as  possible.  The  Create  Text  task 
consisted  of  participants  being  presented  five  words,  with  each  word  being  five  characters 
long.  They  were  asked  to  type  the  presented  text  into  a  box  that  appeared  above  the 
words.  The  Read  Text  tasks  entailed  participants  to  read  ten  words  that  appeared  in  a  box. 
Each  word  was  five  characters  in  length.  The  Pan  and  Zoom  tasks  required  the  volunteers 
to  navigate  the  touchscreen  icons  as  per  the  requested  instructions.  Participants  could  Pan 
'left',  'right',  'up'  or  'down',  and  zoom  either  'in'  or  'out'. 
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The  Read  Unit  tasks  required  participants  to  read  out  loud  the  type,  size  and  location 
coordinates  of  a  new  enemy  icon  that  appeared  on  the  screen  (Appendix  G).  There  were 
five  enemy  types,  and  three  sizes  for  the  enemy  units  that  could  have  appeared.  The 
Create  Unit  tasks  involved  participants  interacting  with  the  touchscreen  to  place  an  icon  at 
the  instructed  location.  In  the  Create  Line  tasks  participants  needed  to  place  their  finger  on 
the  touchscreen  at  a  specified  grid  location.  They  then  needed  to  follow  the  instructions 
and  move  one  grid  unit  either  'up',  'down',  'left',  or  'right'  and  place  their  finger  on  that 
location.  They  were  required  to  repeat  this  a  further  three  times. 


4.  Results 

4.1  Motion  Sickness  Questionnaires 

4.1.1  Motion  Sickness  Susceptibility  Questionnaire  (MSSQ) 

The  motion  sickness  susceptibility  questionnaire  was  administered  to  determine  whether 
participants  were  susceptible  to  motion  sickness.  If  participants  were  found  to  be 
susceptible  to  motion  sickness,  they  were  advised  to  re-consider  participating  in  the 
experiment.  The  mean  MSSQ  score  was  9.21  and  the  standard  deviation  was  9.20. 

It  is  considered  that  MSSQ  scores  above  22.8  could  indicate  that  a  person  is  susceptible  to 
motion  sickness  (Golding,  2006).  One  participant  had  scored  above  22.8,  however  the 
participant  was  willing  to  continue  with  the  experiment. 

4.1.2  Simulator  Sickness  Questionnaire 

The  simulator  sickness  questionnaire  was  used  to  determine  if  participants  became  sick 
after  being  exposed  to  the  simulator.  Below  is  a  chart  of  the  responses  of  the  questionnaire 
comparing  pre-exposure  to  motion  versus  post  "level  two"  and  "level  four"  motion. 


Figure  5:  Average  SSQ  responses  which  include  data  for  all  sessions.  Error  bars  represent  +/-  1 
standard  error  (SE). 

Figure  5  shows  that  participants  reported  feeling  worse  after  "level  4"  motion  exposure 
compared  to  before  motion  exposure  (tis  =  3.112,  p  <  .05),  but  they  did  not  report  feeling 
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worse  after  "level  2"  motion  versus  before  motion  exposure  (tis  =  .908,  p  >  .05).  However, 
no  participants  wished  to  discontinue  as  a  result  of  simulator  sickness. 

Since  participants  reported  being  more  sick  after  being  exposed  to  the  "level  4"  motion,  a 
regression  analysis  was  done  to  see  if  this  correlated  with  performance.  It  was  found  that 
there  was  no  performance  decrements  due  to  increases  in  SSQ  scores  (R2  =  .006,  F13  =  .190, 
p  >  .05). 

4.2  BMS  Performance 

4.2.1  Overall  Analyses  Averaged  Across  Sessions 

All  scores  were  converted  to  percentages  and  inputted  into  the  SPSS  18.0.0  statistical 
package.  A  (3  x  6)  repeated  measures  analysis  of  variance  (ANOVA)  was  conducted  at  the 
0.05  level.  Partial  eta-squared  values  (qp2)  are  provided  as  estimators  of  effect  sizes  for  the 
ANOVAs.  The  figure  below  shows  this  data  graphically. 


Figure  6:  BMS  Task  Performance  scores.  Error  bars  represent  +/~  1  standard  error  (SE). 

As  expected  the  two-way  repeated  measures  ANOVA  revealed  significant  main  effects  of 
motion  (F246  =  62.05,  p  <  .05,  qp2  =  .81),  and  task  type  (F546  =  35.71,  p  <  .05,  qp2  =  .70).  It  also 
yielded  significant  interactions  between  motion  and  task  type  (F10/16  =  15.51,  p  <  .05,  qp2  = 
.51). 

4.2.2  Motion  Condition 

Further  post  hoc  analyses  were  done  using  the  Scheffe  procedure.  Due  to  the  main  effect  of 
motion  in  the  overall  analysis,  contrasts  were  conducted  to  see  exactly  where  the 
differences  in  the  motion  conditions  were.  There  was  no  difference  between  the  static 
motion  (M  =  97.00,  SD  =  1.25)  versus  the  mild  motion  (M  =  95.79,  SD  =  1.73)  conditions 
(Ft/16  =  6.17,  p  >  .05,  qp2  =  .29).  There  was  a  statistically  significant  difference  between  the 
mild  (M  =  95.79,  SD  =  1.73)  versus  the  high  (M  =  87.91,  SD  =  3.44)  motion  conditions  (Fi,i6 
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=  95.56,  p  <  .05,  qp2  =  .86).  It  was  found  that  participant's  performances  were  worse  in  the 
high  versus  the  mild  motion.  This  is  depicted  in  the  Figure  7. 


Figure  7.  Performance  scores  of  each  motion  condition  averaged  across  all  task  types.  Error  bars 
represent  +/- 1  standard  error  (SE). 


4.2.3  Task  Type 

The  significant  main  effect  of  task  type  was  analysed  further  in  a  series  of  contrasts.  This 
was  to  observe  which  tasks  participants  found  easier  than  others.  A  bar  chart  of  the  task 
performances  can  be  seen  in  Figure  8.  Firstly  a  comparison  between  the  visual  tasks  (Read 
Text  and  Read  Unit)  (M  =  99.24,  SD  =  1.16)  and  non  visual  tasks  (Pan  and  Zoom,  Create 
Text,  Create  Unit,  and  Create  Line)  (M  =  91.89,  SD  =  10.19)  was  done.  Participants 
performed  better  in  the  visual  tasks  compared  to  the  non  visual  tasks. 

However,  as  shown  in  Figure  8,  the  Create  Text,  and  Create  Line  scores  were  lower  than 
all  other  tasks  (Pan  and  Zoom,  Read  Text,  Read  Unit,  and  Create  Unit).  The  difference 
between  Create  Text  and  Create  Line  (M  =  86.59,  SD  =  11.70)  versus  the  other  tasks  (Pan 
and  Zoom,  Read  Text,  Read  Unit,  Create  Unit)  (M  =  98.21,  SD  =  3.04)  was  statistically 
significant  (Fi,i6  =  110.91,  p  <  .05,  qp2  =  .88).  Thus  participants  performed  better  in  all  other 
tasks  compared  to  the  Create  Text  and  Create  Line  tasks.  Furthermore,  the  difference 
between  Create  Text  (M  =  83.85,  SD  =  13.09)  and  Create  Line  (M  =  89.33,  SD  =  9.48)  task 
performance  was  not  statistically  significant  (F i,i6  =  4.011,  p  >  .05,  qp2  =  .21).  Therefore  the 
results  show  that  the  Create  Text  and  Create  Line  tasks  were  equally  difficult. 

A  comparison  between  Read  Text  and  Read  Unit  (M  =  99.24,  SD  =  1.16)  versus  Pan  and 
Zoom  (M  =  98.71,  SD  =  1.57)  showed  no  statistical  difference  (Fi,i6  =  4.89,  p  >  .05,  qp2  =  .25). 
Thus  the  performances  in  those  three  tasks  were  similar  to  each  other.  That  is,  participants 
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performed  equally  well  in  the  Pan  and  Zoom,  Read  Text  and  Read  Unit  tasks  averaged 
across  all  three  motion  conditions. 


Figure  8.  Performance  scores  of  each  task  types  averaged  across  all  motion  conditions.  Error  bars 
represent  +/- 1  standard  error  (SE). 

4.2.4  Interaction 

The  significant  interaction  found  in  the  main  ANOVA  was  further  explored.  No 
interaction  between  the  static  versus  mild  and  tasks  was  found. 

The  difference  between  static  Pan  and  Zoom  (M  =  98.44,  SD  =  1.61),  mild  Pan  and  Zoom 
(M  =  98.83,  SD  =  1.86),  and  high  Pan  and  Zoom  (M  =  98.68,  SD  =  1.50),  performance  scores 
were  not  statistically  significant  (p2,i6  =  -20,  p  >  .05,  qp2  =  .01).  Similarly,  no  differences 
were  found  between  static  Read  Text  (M  =  99.64,  SD  =  .50),  mild  Read  Text  (M  =  99.32,  SD 
=  .72),  and  high  Read  Text  (M  =  98.75,  SD  =  .70)  performances  (p2,i6  =  13.36,  p  >  .05,  qp2  = 
.47),  or  static  Read  Unit  (M  =  99.29,  SD  =  .90),  mild  Read  Unit  (M  =  99.48,  SD  =  .86),  and 
high  Read  Unit  (M  =  98.44,  SD  =  1.82)  performances  (F2,i6  =  2.81,  p  >  .05,  qp2  =  .16)  tasks. 
Surprisingly,  there  was  no  statistical  difference  between  the  static  Create  Text  (M  =  90.94, 
SD  =  4.33),  mild  Create  Text  (M  =  85.75,  SD  =  8.46),  and  high  Create  Text  (M  =  74.45,  SD  = 
17.22)  performance  scores  (F2,i6  =  17.45,  p  >  .05,  qp2  =  .54).  The  lack  of  statistical  difference 
between  motion  conditions  for  the  Create  Text  tasks  was  likely  due  to  the  high  variability 
in  the  data. 

There  was  a  significant  difference  between  the  static  Create  Line  (M  =  94.81,  SD  =  4.41), 
mild  Create  Line  (M  =  93.82,  SD  =  4.19)  and  high  Create  Line  (M  =  79.25,  SD  =  9.01)  scores 
,i6  =  39.08,  p  <  .05,  qp2  =  .72).  There  was  also  a  significant  difference  between  the  static 
Create  Unit  (M  =  98.96,  SD  =  1.52),  mild  Create  Unit  (M  =  97.53,  SD  =  2.18),  and  high 
Create  Unit  (M  =  90.18,  SD  =  4.50)  scores  (p2,i6  =  45.27,  p  <  .05,  qp2  =  .75). 
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4.3  Learning  Effects 

A  regression  analysis  was  performed  at  the  0.05  level  to  determine  if  there  were  any 
learning  effects  throughout  the  six  sessions  for  the  different  motion  conditions.  The  figure 
below  shows  the  average  performances  and  trend  lines  for  all  tasks  in  the  different  motion 
conditions  over  the  six  sessions. 


Figure  9.  Average  performance  and  trend  line  using  all  tasks  for  each  session  under  the  different 
motion  conditions.  Error  bars  represent  +/- 1  standard  error  (SE). 


It  was  found  that  participant  performance  improved  during  the  initial  training  sessions  (R2 
=  .013,  Fi/574  =  7.447,  p  <  .05).  Once  trained  however,  their  performance  during  the  static 
sessions  remained  the  same  (R2  =  .001,  Fi^s  =  .524,  p  >  .05).  There  continued  to  be  an 
improvement  for  both  the  mild  (R2  =  .008,  Fi;%8  =  4.450,  p  <  .05)  and  high  (R2  =  .009,  Fi,568  = 
5.342,  p  <  .05)  motion  conditions  across  the  six  sessions. 

4.3.1  Individual  Tasks 

Since  there  were  differences  between  the  performances  for  the  task  types,  further  analysis 
was  done  to  determine  if  participants  learnt  the  tasks  differently. 

4. 3. 1.1  Pan  and  Zoom 

There  was  no  difference  in  performance  between  the  training  (R2  =  .004,  Fi,94  =  .424,  p  > 
.05),  static  (R2  =  .003,  Fi,93  =  .003,  p  >  .05),  mild  (R2  =  .000,  Fi^  =  .003,  p  >  .05)  or  high 
motion  conditions  (R2  =  .011,  Fi,93  =  1.033,  p  >  .05).  That  is,  participants  performed  just  as 
well  across  all  the  motion  conditions  for  the  Pan  and  Zoom  task.  This  can  be  seen  in  the 
figure  below. 
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Figure  10.  Average  performance  and  regression  line  for  the  Pan  and  zoom  task  across  the  six 
sessions.  Error  bars  represent  +/- 1  standard  error  (SE). 


4.3. 1.2  Read  Text 

Performance  in  the  Read  Text  task  statistically  improved  for  the  training  sessions  (R2  = 
.009,  Fi,94  =  .841,  p  >  .05),  however,  this  improvement  would  be  considered  operationally 
insignificant.  Performance  did  not  change  across  the  sessions  for  the  other  conditions, 
static  (R2  =  .009,  Fi,93  =  .841,  p  >  .05),  mild  (R2  =  .026,  Fi,93  =  2.486,  p  >  .05)  or  high  sessions 
(R2  =  .000,  Fi,93  =  .005,  p  >  .05).  This  is  shown  in  the  figure  below. 
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Figure  11.  Performance  and  regression  lines  across  the  six  sessions  for  the  training,  static,  mild  and 
high  conditions.  Error  bars  represent  +/- 1  standard  error  (SE). 
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4.3. 1.3  Read  Unit 

Performance  in  the  Read  Unit  task  statistically  improved  for  the  training  sessions  (R2  = 
.050,  Fi,94  =  4.913,  p  <  .05),  however,  this  improvement  would  be  considered  operationally 
insignificant.  Performance  did  not  improve  for  the  other  conditions,  static  (R2  =  .007,  Fi,93  = 
.671,  p  >  .05),  mild  (R2  =  .017,  F1/93  =  1.637,  p  >  .05)  and  high  (R2  =  .003,  Fi,93  =  .256,  p  >  .05), 
across  the  sessions.  This  is  depicted  in  the  figure  below. 
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Figure  12.  Regression  lines  and  average  performance  for  the  Read  Unit  task  over  the  six  sessions. 
Error  bars  represent  +/- 1  standard  error  (SE). 


4.3. 1.4  Create  Text 

Participants  showed  no  significant  improvement  for  the  Create  Text  task  for  either  the 
training  (R2  =  .030,  Fi,94  =  .841,  p  >  .05)  or  the  static  (R2  =  .001,  Fi,93  =  .086,  p  >  .05) 
conditions.  However,  the  performance  in  the  Create  Text  task  did  improve  for  the  mild  (R2 
=  .089,  Fi/93  =  9.063,  p  <  .05)  and  high  (R2  =  .040,  Fi/93  =  3.888,  p  <  .05)  conditions  over  the  six 
sessions. 
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Figure  13.  Create  Text  average  performance  and  regression  lines  for  the  six  sessions.  Error  bars 
represent  +/- 1  standard  error  (SE). 


4.3. 1.5  Create  Line 

The  only  statistically  significant  performance  improvement  for  the  Create  Line  task  across 
the  different  sessions  was  for  the  high  motion  condition  (R2  =  .043,  Fi,93  =  4.162,  p  <  .05). 
None  of  the  other  conditions  training  (R2  =  .014,  Fi/m  =  1.364,  p  >  .05),  static  (R2  =  .009,  Fi,93 
=  .886,  p  >  .05)  or  mild  (R2  =  .005,  Fi,93  =  .470,  p  >  .05)  yielded  performance  improvements 
over  the  sessions. 


Figure  14.  Average  performance  and  regression  lines  for  the  Create  Line  tasks  across  the  six 
sessions.  Error  bars  represent  +/- 1  standard  error  (SE). 
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4.3. 1.6  Create  Unit 

The  only  condition  is  which  performance  improved  over  the  six  sessions  for  the  Create 
Unit  task  was  during  the  training  (R2  =  .062,  Fi,94  =  .841,  p  <  .05).  In  all  other  conditions, 
static  (R2  =  .026,  Fi,93  =  2.476,  p  >  .05),  mild  (R2  =  .040,  Fi,93  =  3.855,  p  >  .05),  and  high  (R2  = 
.012,  Fi,93  =  1.083,  p  >  .05),  the  performance  did  not  improve  over  the  sessions. 


Figure  15.  Regression  lines  and  average  performance  for  the  Create  Unit  task  over  the  six  sessions. 
Error  bars  represent  +/- 1  standard  error  (SE). 


4.4  Subjective  assessments 

4.4.1  NASA  Raw  Task  Load  Index 

Figure  16  shows  the  overall  RTLX  responses  for  each  motion  condition.  The  chart  shows  a 
box  plot  where  the  central  line  indicates  the  median  value,  the  inner  box  indicates  the  +/- 
25%  values,  and  the  outer  whiskers  indicate  the  outer  +/-  25%  values.  The  difference 
between  the  mean  and  median  for  the  static  condition  shows  the  data  being  skewed.  There 
was  no  significant  difference  between  the  static  (mean=26,  SD=16)  and  motion  "Level 
two"  (mean=32,  SD=16),  and  a  significant  difference  between  the  static  and  motion 
"Level  four"  (mean=56,  SD=20),  as  well  as  between  motion  "Level  two"  and  "Level  four". 
Thus  both  the  static  and  "Level  two"  conditions  were  rated  as  "low"  workload  and  "Level 
four"  as  half  way  between  "low"  and  "high". 
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Figure  16:  Subjective  RTLX_Mental  Demand%  vs  Motion  Level 


Figure  17  shows  the  subjective  Raw  Task  Load  Index  (RTLX)  responses  for  all  sessions  vs 
the  motion  conditions. 
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Figure  17:  Subjective  RTLX  vs  motion  level  +/-  1  standard  error  (SE). 

4.4.2  BMS  Task  Difficulty 

Figure  18  shows  the  overall  (normalised)  subjective  BMS  Task  Difficulty  for  the  three 
motion  conditions.  The  value  20  represents  the  minimum  value  of  "not  difficult".  The 
value  of  100  represents  the  maximum  value  of  "extremely  difficult".  The  value  of  60 
represents  the  half-way  value.  The  chart  shows  a  box  plot  where  the  central  line  indicates 
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the  median  value,  the  inner  box  indicates  the  +/-  25%  values,  and  the  outer  whiskers 
indicate  the  outer  +/-  25%  values.  In  the  motion2  condition  there  are  three  outlier  values. 
The  difference  between  the  mean  and  median  for  the  "level  four"  condition  shows  the  data 
being  skewed.  There  was  a  significant  difference  between  the  static  (mean=28,  SD=7)  and 
"Level  two"  motion  (mean=34,  SD=8),  and  a  significant  difference  between  the  static  and 
"Level  four"  motion  (mean=63,  SD=15).  There  was  also  a  significant  difference  between 
"Level  two"  and  "Level  four"  motions.  Note  that  the  normalised  subjective  difficulty 
average  of  63  for  the  "Level  four"  motion  is  half  way  between  "not  difficult  "  and 
"extremely  difficult". 
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Figure  18:  Task  Difficulty  vs.  Motion  Condition 

Figure  19  shows  the  BMS  Task  Difficulty  vs  Motion  Level  for  the  individual  tasks.  Note 
that  the  normalised  means  for  Static:  Pan_Zoom,  ReadJUnit,  and  Reading  tasks  were  20 
and  thus  are  difficult  to  see  in  the  figure. 

It  is  interesting  to  note  that  the  reading  task  under  "Level  four"  motion  was  considered  to 
be  almost  as  difficult  as  the  creating  text,  units  and  lines  tasks,  but  this  was  not  reflected  in 
actual  performance. 
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Figure  19:  BMS  Subjective  Task  Difficulty  vs  Motion  Level 


5.  Discussion 


5.1  Motion  Condition 

The  results  of  this  experiment  support  the  first  hypothesis  as  no  overall  difference  between 
the  static  and  mild  motion  conditions  were  observed.  However  there  was  a  difference 
between  the  mild  and  high  motions  (as  shown  in  Figure  8).  Therefore,  commanders  could 
be  expected  to  relay  information  through  the  BMS  as  well  as  receive  information  from  the 
BMS  "on  the  move"  if  the  motion  is  relatively  small.  More  specifically,  if  the  motion  is 
lower  than  an  RMS  value  of  0.47m/s2,  commanders  should  not  have  problems  entering  as 
well  as  reading  information  through  the  BMS. 

However,  if  the  motion  is  high,  then  commanders  would  be  able  to  receive  information, 
but  it  would  be  a  lot  more  difficult  to  send  (input)  information.  This  study  showed  that  for 
an  RMS  value  above  1.2m/  s2,  commanders  could  not  send  information  through  the  BMS 
as  well  as  during  the  mild  motion  conditions.  Further  research  is  needed  to  ascertain  what 
motion  level  between  0.47m/s2  and  1.2m/s2,  is  likely  to  lead  to  a  statistically  different 
result  from  the  mild  motion  condition.  This  issue  can  be  further  investigated  with  DSTO's 
new  LAMP  capability  described  below. 

The  experiment  also  supports  previous  studies'  findings  (e.g.  Metcalfe  et  al  2008)  that 
performance  degrades  with  higher  levels  of  motion.  However,  this  study  extends  previous 
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literature  as  it  has  used  a  three  degree  of  freedom  (multi-axis)  simulator  to  generate  the 
motion.  As  Seagull  &  Wickens'  (2006)  review  of  the  literature  had  shown,  most  studies 
simply  used  z-axis  (vertical,  heave)  motion  to  evaluate  task  performance.  Salmon  et  al 
(2010)  also  comments  on  the  ambiguity  of  such  studies  when  trying  to  evaluate  the  effect 
of  different  motion  levels  on  BMS  performance.  The  three  degree  of  freedom  simulator 
added  two  other  axes  of  motion  than  a  simple  up  and  down  motion  simulation  and 
arguably,  this  is  more  realistic  of  what  military  crew  would  experience.  Therefore,  a  more 
applicable  motion  profile  was  used  to  validate  the  results. 

5.2  BMS  Performance 

These  results  also  support  hypothesis  two,  as  there  is  a  clear  difference  in  performance 
based  on  the  type  of  task  that  was  performed,  and  it  also  extends  previous  literature  in  this 
area.  As  Salmon  (2010)  noted,  previous  field  research  using  a  BMS  system  under  motion 
has  not  measured  specific  task  related  performance.  Rather,  it  has  focused  on  issues  such 
as  reach  timing  and  accuracy.  This  study  shows  that  even  after  a  person  has  'reached'  the 
BMS  system,  there  will  be  differences  in  accuracy  depending  on  the  type  of  task  at  hand. 
Furthermore,  as  Zywiol  et  al  (2006)  has  shown,  lab  experiments  in  this  area  have  also 
concentrated  on  issues  such  as  reach  timing  and  accuracy,  and  were  concerned  with 
making  user  interface  recommendations.  Furthermore,  previous  laboratory  based  studies 
did  not  focus  on  the  operational  performance  of  BMS  tasks  under  motion,  as  this  study 
has. 

Figure  6  shows  that  participants  found  the  Read  Text,  Read  Unit  and  Pan  and  Zoom  tasks 
are  relatively  easy,  as  they  performed  above  98%  even  in  the  high  motion  condition. 
However,  the  Create  Text,  Create  Unit  and  Create  Line  task  performances  show  that  they 
are  clearly  more  difficult  than  the  other  three  tasks.  Furthermore,  the  performance  for  the 
Create  Text,  Create  Unit  and  Create  Line  tasks  degraded  as  the  motion  level  increased. 
Thus  there  is  a  clear  distinction  between  the  reading  and  simple  button  pressing  tasks 
versus  the  creating  tasks.  The  reading  task  is  a  type  of  task  which  feeds  information  to  a 
commander.  The  creating  tasks  occur  when  a  commander  gives  information  to  others.  It 
can  therefore  be  inferred  that,  under  high  motion  conditions,  it  would  be  appropriate  to 
give  information  to  a  commander  and  ask  them  to  perform  very  simple  button  presses 
(Pan  and  Zoom),  rather  than  enter  more  difficult  textual  or  graphical  information. 

5.3  Learning  Effects 

The  study  supported  the  third  hypothesis  that  there  is  a  learning  effect  depending  on  the 
type  of  task  as  well  as  the  type  of  motion.  Performance  improved  across  the  training,  mild 
and  high  sessions,  but  not  fore  the  static  condition.  Noting  that  the  training  was  under  no 
motion,  this  result  indicates  that  participants  had  to  mitigate  against  the  new  motion 
stimuli.  Essentially  participants  appeared  to  be  learning  how  to  cope  with  the  motion. 
Hence,  performance  in  the  mild  and  high  conditions  improved  over  the  sessions. 

Performance  in  the  training  sessions  only  improved  for  the  Read  Text,  Read  Unit  and 
Create  Unit  tasks.  Performance  did  not  improve  in  the  training  session  for  the  Pan  and 
Zoom,  Create  Text,  and  Create  Line  tasks.  This  shows  that  some  tasks  do  not  need  as  much 
training.  In  particular,  in  the  Pan  and  Zoom,  Create  Unit,  Read  Text,  and  Read  Unit  tasks 
there  was  no  learning  effect  for  the  static,  mild  or  high  motion  conditions.  That  is. 
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performance  remained  the  same  across  the  six  experimental  sessions  across  each  of  the 
motion  conditions.  However,  performance  did  improve  for  the  Create  Text  and  the  Create 
Line  tasks  in  the  high  motion  condition.  Furthermore,  performance  improved  in  the  mild 
motion  condition  for  the  Create  Text  task. 

This  shows  that  it  would  be  valuable  to  train  crew  under  motion,  especially  for  the  more 
difficult  output  tasks  such  as  creating  text  or  creating  line  type  of  tasks.  Lenne  et  al  (2010) 
suggested  that,  in  general,  tasks  are  best  learnt  in  realistic  conditions.  This  study  has 
reinforced  that  suggestion. 

5.4  Subjective  Assessments 

Subjects  generally  rated  the  "Level  four"  motion  as  requiring  high  effort  and  difficult. 
There  was  little  difference  between  ratings  for  the  Static  and  "Level  two"  motion. 

The  SSQ  scores  were  higher  for  the  "Level  four"  motion  compared  to  the  pre  motion 
condition.  However,  this  did  not  lead  to  performance  decrements.  Therefore,  the 
performance  decrements  found  above  were  due  to  increases  in  motion  magnitude,  rather 
than  simulator  sickness. 

The  subjective  questionnaire  responses  presented  in  Appendix  I  provide  information 
about  learning  effects.  Subjects  found  Reading  and  Pan-Zoom  particularly  easy  with  no 
learning  curve.  Writing,  Create-Line,  and  Create-Unit  appeared  to  require  learning  time 
across  a  number  of  sessions. 

5.5  Limitations 

Despite  best  efforts,  participant  scheduling  was  not  consistent  throughout  the  whole 
experiment.  This  was  because  unforseen  events  or  meetings  continuously  came  up  over 
the  six  weeks,  which  participants  could  not  avoid.  However,  there  was  always  a  minimum 
of  three  days  in  between  sessions  and  a  maximum  of  ten  days. 

Also,  the  relatively  small  number  of  participants  means  that  the  results  have  to  be  taken 
with  certain  caution.  For  instance,  some  participants  may  have  accidentally  pressed  the 
touchscreen  in  the  'Create  Line'  task  in  an  unintended  location,  leading  to  an  exaggerated 
decrease  in  performance.  With  a  small  number  of  participants,  these  errors  become 
amplified,  which  may  lead  to  inaccurate  results. 

The  limited  displacement  of  the  three  degree  of  freedom  motion  platform  limited  the  type 
of  motion  that  could  be  subjected  to  the  participants.  Although  the  produced  level  of 
vibrations  used  for  the  experiment  aligned  with  the  "level  two"  and  "level  four"  ranges 
according  the  standards,  it  is  not  indicative  of  the  type  of  motion  that  might  occur  in  the 
real  world.  A  simulator  with  larger  actuator  arms  may  have  been  able  to  produce  more 
realistic  motion  profiles. 
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5.6  Future  and  Other  Research 

A  field  study  was  also  carried  out  with  light  armoured  field  vehicles.  The  vehicles  were 
driven  over  various  terrains,  and  the  vibration  measurements  were  taken  for  each  drive. 
There  were  three  main  types  of  road  that  the  vehicles  were  driving  over,  smooth  paved 
roads,  gravel  road  and  rough  cross  country  roads.  The  participant  sitting  in  the 
commander  position  completed  the  BMS  tasks  as  described  above.  Analyses  of  the  road 
vibrations  have  been  done.  The  maximum  frequency  weighted  acceleration  attained  from 
all  the  drives  was  1.42m/ s2.  The  results  of  the  BMS  tasks  will  be  matched  against  the 
frequency  weighted  accelerations  in  order  to  determine  performance  at  the  different 
vibration  levels. 

The  field  study  also  showed  that  the  maximum  frequency  weighted  acceleration  was 
higher  than  the  one  used  in  the  lab  study.  This  means  that  military  drivers  often  exceed 
the  highest  vibration  level  than  given  in  this  lab  study.  The  field  study  also  had  large  crest 
factors,  indicating  that  there  were  more  dynamic  motion  profiles.  Future  studies,  with 
larger  motion  capabilities  will  need  to  incorporate  higher  levels  of  motion  as  one  of  its 
conditions. 

DSTO's  motion  platform  simulation  capability  program  will  culminate  with  the 
commissioning  of  DSTO's  six  degree  of  freedom  LAMP  later  this  year.  The  LAMP  will 
allow  research  into  the  effect  of  much  greater  ranges  of  motion  on  BMS  performance.  It  is 
anticipated  that  with  a  greater  range  on  motion  simulation  capability,  there  will  be  a 
greater  ability  to  re-create  more  realistic  and  ecologically  valid  simulations.  Llowever,  this 
is  to  be  verified  through  validation  exercises  and  experiments.  The  current  experiment  will 
be  replicated  using  the  LAMP,  in  order  to  verify  the  current  results  and  also  to  rectify 
some  of  the  limitations  of  the  current  study.  This  will  be  followed  by  further  experiments 
aimed  to  identify  the  levels  of  motion  beyond  which  even  pull  tasks  (i.e.  reading  text  and 
icons)  becomes  difficult. 


6.  Conclusion 


The  study  found  that  participants  could  cope  with  a  mild  level  of  motion,  but  performance 
degraded  when  the  motion  was  high.  More  specifically,  participants  showed  no  difference 
in  performance  between  motions  less  than  0.47m/ s2  and  a  static  motion  situation. 
Llowever,  there  was  a  difference  between  the  mild  and  high  motion  condition.  It  also 
found  that  there  was  a  difference  in  performance  depending  on  the  type  of  tasks  the 
participants  were  performing.  In  particular,  it  was  more  difficult  to  enter  information 
compared  to  reading  information.  It  appears  easier  to  'push'  information  to  participants, 
rather  than  'pull'  information  from  them.  Lastly,  there  were  learning  effects  found  for  both 
motion  and  task  type.  There  was  no  learning  effect  for  the  static  condition,  but  there  were 
learning  effects  for  both  the  mild  and  high  conditions.  Learning  effects  were  apparent 
during  the  Create  Line  and  Create  Text  tasks  in  the  high  motion  condition.  The  Create  Text 
tasks  also  showed  learning  effects  in  the  mild  motion.  None  of  the  other  tasks  showed 
learning  effects  in  the  experimental  conditions  (static,  mild  or  high).  This  implies  that  BMS 
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training  for  use  in  vehicles  should  include  motion  conditions  so  users  learn  how  to  adapt 
to  vehicle  motion. 

This  study  improved  upon  previous  research  looking  at  performance  degradation  due  to 
motion.  It  used  technology  which  has  not  been  used  widely  before,  by  employing  a  three 
degree  of  freedom  motion  simulator.  By  employing  such  methods,  it  created  a  more 
realistic  motion  cue,  closer  to  what  military  personnel  might  experience  in  the  field.  This 
gives  the  results  more  ecological  validity.  However,  further  research  needs  to  be  done  in 
this  area,  to  fully  understand  how  larger  motion  affects  performance  in  a  military  context. 
DSTO's  new  LAMP  capability  will  aid  in  exploring  further  research  questions  in  this  area. 
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Appendix  A:  Motion  Sickness  Susceptibility 

Questionnaire 


Short  Motion  Sickness  Susceptibility  Questionnaire 

PARTICPANT  ID: _ 

Please  give  your  answers  on  the  dotted  lines,  or  encircle  one  of  the  printed  options 


Have  you  ever  had  any  medical  conditions  involving  your  ears?  Yes  No 

■  If  yes,  what  conditions  did  you  have? . 

■  And  how  old  were  you?  . years 

Do  you  suffer  from  headaches?  Never  /  seldom  /  sometimes  /  often 

■  If  yes,  did  your  doctor  characterise  this  as  migraine?  Yes  No 

The  next  questions  refer  to  your  sensitivity  to  motion  sickness  in  the  past  and  to  the  kind 
of  motions  that  you  dislike  most.  Here,  motion  sickness  refers  to  a  clear  feeling  of 
discomfort,  nausea,  or  vomiting  due  to  motion. 


How  often  did  you  feel  sick  as  a  child  (below  the  age  of  12  years)  in 

cars  n.a  /  never  /  seldom  /  sometimes  /  often 

busses  n.a  /  never  /  seldom  /  sometimes  / often 

trains  n.a  /  never  /  seldom  /  sometimes  / often 


aircraft 
small  boats 
large  ships 


n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  /  often 
n.a  /  never  /  seldom  /  sometimes  / often 


n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 


swings 

merry-go-rounds 
fair  ground 

attractions 

Did  you  ever  have  to  throw  up  because  of  motion  sickness  as  a  child? 


Yes 


No 


How  often  did  you  feel  sick  in  the  past  12  years  in 

cars  n.a  /  never  /  seldom  /  sometimes  / often 

busses  n.a  /  never  /  seldom  /  sometimes  / often 

trains  n.a  /  never  /  seldom  /  sometimes  / often 


aircraft 
small  boats 
large  ships 


n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 


swings 

merry-go-rounds 
fair  ground  attractions 


n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 
n.a  /  never  /  seldom  /  sometimes  / often 


Did  you  ever  have  to  throw  up  because  of  motion  sickness  in  the  past  12  years 

Yes  No 
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Appendix  B:  Simulator  Sickness  Questionnaire 

Simulator  Sickness  Questionnaire 
Symptom  Checklist 

PARTICPANT  ID: _ CONDITION: _  SESSION: _ 

PRE-Simulator 

Circle  below  if  any  of  the  symptoms  apply  to  you  now 

1.  General  discomfort  none  slight  moderate  severe 

2.  Fatigue  none  slight  moderate  severe 

3.  Headache  none  slight  moderate  severe 

4.  Eyestrain  none  slight  moderate  severe 

5.  Difficulty  focusing  none  slight  moderate  severe 

6.  Increased  salivation  none  slight  moderate  severe 

7.  Sweating  none  slight  moderate  severe 

8.  Nausea  none  slight  moderate  severe 

9.  Difficulty  concentrating  none  slight  moderate  severe 

10.  Fullness  of  the  head  none  slight  moderate  severe 

11.  Blurred  vision  none  slight  moderate  severe 

_ Please  stand  up  when  assessing  for  symptoms  12  -  14 _ 

12.  Dizziness  (eyes  open)  none  slight  moderate  severe 

13.  Dizziness  (eyes  closed)  none  slight  moderate  severe 

14.  Vertigo  none  slight  moderate  severe 

15.  Stomach  awareness  none  slight  moderate  severe 

16.  Burping  none  slight  moderate  severe 
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Simulator  Sickness  Questionnaire 
Symptom  Checklist 

PARTICPANT  ID: _  CONDITION: _  SESSION: _ 


POST-Simulator 


Circle  below  if  any  of  the  symptoms  apply  to  you  now 


1.  General  discomfort  none  slight  moderate  severe 

2.  Fatigue  none  slight  moderate  severe 

3.  Headache  none  slight  moderate  severe 

4.  Eyestrain  none  slight  moderate  severe 

5.  Difficulty  focusing  none  slight  moderate  severe 

6.  Increased  salivation  none  slight  moderate  severe 

7.  Sweating  none  slight  moderate  severe 

8.  Nausea  none  slight  moderate  severe 

9.  Difficulty  concentrating  none  slight  moderate  severe 

10.  Fullness  of  the  head  none  slight  moderate  severe 

1 1.  Blurred  vision  none  slight  moderate  severe 


_ Please  stand  up  when  assessing  for  symptoms  12  -  14 _ 

12.  Dizziness  (eyes  open)  none  slight  moderate  severe 

13.  Dizziness  (eyes  closed)  none  slight  moderate  severe 

14.  Vertigo  none  slight  moderate  severe 


15.  Stomach  awareness  none  slight  moderate  severe 

16.  Burping  none  slight  moderate  severe 
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Appendix  C:  NASA-TLX 

NASA-Task  Load  Index 

PARTICPANT  ID: _  CONDITION: _  SESSION: 


Please  rate  your  workload  by  marking  a  box  on  each  of  the  seven  scales,  which  matches 

your  experience. 

Mental  Demand 


r 

1 

1 

Low 

Physical  Demand 

High 

l 

1 

1 

Low 

Temporal  Demand 

High 

l 

1 

1 

Low 

Performance 

High 

l 

1 

1 

Good 

Effort 

Poor 

l 

1 

1 

Low 

Frustration 

High 

l 

1 

1 

Low 

High 
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NASA-Task  Load  Index  Rating  Scale  Definitions 


Title 

Endpoints 

Descriptions 

Mental  Demand 

Low/ High 

How  much  mental  and  perceptual  activity 
was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking,  searching, 
etc.)?  Was  the  task  easy  or  demanding,  simple 
or  complex,  exacting  or  forgiving? 

Physical 

Demand 

Low/ High 

How  much  physical  activity  was  required 
(e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious? 

Temporal 

Demand 

Low/  High 

How  much  time  pressure  did  you  feel  due  to 
the  rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 

Performance 

Good/ Poor 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied 
were  you  with  your  performance  in 
accomplishing  these  goals? 

Effort 

Low/ High 

How  hard  did  you  have  to  work  (mentally 
and  physically)  to  accomplish  your  level  of 
performance? 

Frustration 

Level 

Low/ High 

How  insecure,  discouraged,  irritated,  stressed 
and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during 
the  task? 
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Appendix  D:  BMS  Task  Difficulty  Questionnaire 


Task  Difficulty  Questionnaire 

Please  indicate  how  difficult  you  found  the  tasks  you  have  just  performed  by  circling  a 
number. 

1  =  not  difficult.  5  =  extremely 
difficult 

Participant  ID 

Session 

Date 

Condition 

Writing  text  1  2  3  4  5 

Reading  Text  1  2  3  4  5 

Pan  and  Zoom  1  2  3  4  5 

Create 

boundary  12345 

Read  Unit  1  2  3  4  5 

Create  unit  1  2  3  4  5 
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Appendix  E:  Demographic  Questionnaire 

Demographic  &  Military  Experience  Questionnaire 
PARTICPANT  ID: _ 

Please  give  your  answers  on  the  dotted  lines,  or  encircle  one  of  the  printed  options 


1. 

Age: . years  old 

2. 

Gender: 

Male 

Female 

3. 

Which  is  your  dominant  hand? 

Right 

Left 

4. 

Do  you  wear  glasses  when  working  on  the  computer? 

Yes 

No 

5. 

Do  you  have  impairment  to  your  colour  vision?) 

Yes 

■  If  yes,  what  type  of  impairment  (if  known)? 


6.  Have  you  any  experience  with  the  military  unit  icons  used  by  the  Australian  Army? 

Yes  No 

■  If  yes,  what  experience  do  you  have? 


7.  Have  you  ever  used  a  digital  command  and  control  system? 

Yes  No 

■  If  yes,  what  type  of  command  and  control  system  have  you  used  and  what 
experience  have  you  had? 
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Appendix  F:  Participant  Instructions. 

Intro  Script 

•  Explain  study  -  Thanks  for  coming  in  today.  Firstly,  have  a  read  through  the 
Information  Sheet  for  this  study  and  the  Guidelines  for  Volunteers,  and  then  I  will 
give  you  a  demonstration  of  the  tasks  and  the  motion.  [SHOW  TRAINING  OF  TASKS, 
NOT  IN  DETAIL/ GIVE  MOTION  DEMO] 

•  Data  collection  -  Data  will  be  recorded  in  the  following  ways: 

o  Most  data  on  the  speed  and  accuracy  of  your  performance  of  the  6  BMS  tasks  will 
be  recorded  automatically  by  the  BMS  software.  Some  tasks  however  require 
verbal  responses  from  you  and  these  will  be  recorded  manually  by  the 
experimenter. 

•  If  you  agree  to  participate  in  this  experiment  you  will  be  asked  to  attend  up  to  another 
7,  approximately  half  hour,  experimental  sessions.  If  you  also  agree  to  take  part  in  the 
EEG  testing,  it  will  require  an  extra  half  hour  per  session.  Today  and  in  the  next 
session,  you  will  be  required  to  complete  3,  5-10  minute  practice  trials  where  you  will 
complete  24  randomly  assigned  Battle  Management  System  tasks  without  any  motion. 
In  the  six  subsequent  trials,  you  will  complete  the  tasks  with  and  without  motion. 
There  are  two  levels  of  motion,  excluding  the  static  trails.  One  will  simulate  moderate 
vehicle  motion  and  in  other  will  emulate  the  motions  experienced  when  driving  on  a 
relatively  rough  road. 

•  Show  SSQ  -  Before  beginning  each  motion  session,  you  will  be  required  to  complete  a 
Simulator  Sickness  Questionnaire,  which  will  ask  you  about  your  current  physical 
state.  At  the  end  of  each  group  of  tasks,  you  will  again  need  to  complete  the  Simulator 
Sickness  Questionnaire.  This  is  done  to  assess  whether  the  simulator  has  any  affect  on 
your  physical  state.  If  you  begin  feeling  sick  during  your  time  on  the  simulator 
however,  it  is  important  that  you  let  me  know  so  we  can  stop  the  session  immediately. 

•  Show  NASA-TLX  -  After  you  complete  each  experimental  condition,  you  will  be 
required  to  complete  the  NASA  Task  Load  Index,  which  asks  you  to  rate  the  workload 
experienced  during  each  experimental  condition. 

•  Show  Questionnaire  -  You  will  be  asked  to  complete  a  questionnaire  asking  you  to 
rank  how  hard  you  felt  each  task  was  to  complete. 
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Do  you  agree  to  take  art  in  this  study?  To  take  part  in  this  study,  you  will  need  to  sign 

the  Consent  Form. 

Complete  the  Demographic  &  Military  Experience  questionnaire  -  Could  you  please 
complete  this  questionnaire  asking  some  demographic  questions  about  you  as  well  as 
any  military  experience  you  might  have 

Complete  the  Motion  Sickness  Susceptibility  questionnaire.  Could  you  also  please 
complete  this  questionnaire  about  your  past  history  with  motion  sickness?  This 
information  will  help  us  determine  whether  past  history  is  a  good  predictor  of  the  risk 
of  participants  experiencing  motion  sickness  using  this  experimental  equipment. 
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Training  Session  Script 


•  Show  visual  icon  information  sheet:  Here  is  a  sheet  describing  the  icons  that  will  be 
used  throughout  the  experiment.  [SHOW  READ  ICON  DESCRIP  DOC]  The  icons  have 
been  modified  for  non-military  personnel.  Have  a  read  of  this  sheet  explaining  how  the 
information  about  enemy  detections  is  represented  visually  using  icons  displayed  on 
the  map. 

•  Train  participants  in  the  6  Tasks  now  I  will  train  you  to  perform  the  6  Battle 
Management  System  tasks  that  will  be  used  throughout  this  experiment.  You  are  to 
observe  while  I  demonstrate  a  mock  up  of  a  BMS  on  the  touch  screen  in  front  of  me. 
My  task  instructions  are  written  at  the  top  of  the  screen.  Once  I  have  completed  the 
task  I  will  press  the  next  button  at  the  top  right  of  the  screen.  Note  you  will  always  be 
strapped  in  whilst  doing  the  tasks.  Also  you  are  allowed  to  brace  yourself  on  the 
laptop  mount  if  you  wish. 

•  Pan  and  Zoom  (5  minutes)  -  In  this  task  the  written  instructions  at  the  top  of  the 
screen  will  ask  you  to  pan  and  zoom  the  map  display.  You  should  select  the  pan 
button,  which  is  the  third  button  down  on  the  right  hand  side  of  the  screen  and 
then  select  the  arrow  button  in  the  resulting  dialog  that  matches  the  instruction. 
Then,  based  on  the  written  instruction,  you  should  then  select  either  the  zoom  in 
button  (top  button)  or  the  zoom  out  button  (second  from  top)  according  to  the  text 
presented  at  the  top  of  the  screen.  Y ou  should  do  this  as  quickly  and  as  accurately 
as  possible.  Once  you  have  finished  please  press  the  next  button. 

•  Write  Text  (5  minutes)  -  In  this  task  you  should  type  the  text  presented  at  the  top  of 
the  screen  into  the  message  text  box  presented  in  the  middle  of  the  screen.  You 
should  do  this  as  quickly  and  as  accurately  as  possible.  However,  if  you  make  a 
mistake,  please  continue  on  without  trying  to  correct  for  the  error.  Once  you  have 
finished  typing  the  words  please  press  the  next  button. 

•  Read  Type,  Size  and  location  of  an  enemy  Unit  (5  minutes)  -  In  this  task  the  written 
instructions  at  the  bottom  of  the  screen  will  ask  you  to  locate  a  new  red  enemy  icon 
on  the  screen  and  read  out  aloud  its  type  size  and  location.  Y ou  should  read  out  the 
location  from  the  x  axis  numbers  at  the  top  of  the  map  and  then  from  the  y-axis 
numbers  running  down  the  left  hand  side  of  the  map.  I  will  manually  record  your 
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responses.  You  should  do  this  as  quickly  and  as  accurately  as  possible.  Once  you 
have  finished  reading  out  the  details  please  press  the  next  button. 

Draw  a  new  Unit  Movement  Boundary  (5  minutes)  -  In  this  task  the  written 
instructions  at  the  top  of  the  screen  will  ask  you  to  create  a  series  of  joined  lines  on 
the  map  representing  a  unit's  movement  boundary. (this  is  a  line  the  unit  should 
not  cross  during  an  operation).  To  start  the  line  press  down  on  the  specified  map 
location.  This  should  create  a  blue  point.  Then  press  down  exactly  one  grid  form 
that  point  in  the  direction  indicated  in  the  instructions.  This  will  create  a  new  line. 
Then  press  down  one  grid  away  in  the  next  direction  indicated  until  you  have 
created  four  joined  lines.  You  should  do  this  as  quickly  and  as  accurately  as 
possible.  Note  the  lines  do  not  have  to  join  up  to  form  a  square.  Once  you  have 
finished  creating  the  lines  please  press  the  next  button. 

Create  a  nezv  Unit  Icon  (5  minutes)  -  In  this  task  the  written  instructions  at  the  top 
of  the  screen  will  ask  you  to  create  a  particular  type  of  new  red  enemy  icon  at  a 
certain  X  Y  location  on  the  screen.  To  do  this  you  should  hold  your  figure  down  on 
the  screen  at  the  specified  location  for  2  seconds.  This  will  then  bring  up  a  popup 
menu  allowing  you  to  select  the  required  type  of  enemy  unit.  In  the  popup  menu 
select  "Contact  Reports"  and  then  select  the  appropriate  icon.  You  should  do  this 
as  quickly  and  as  accurately  as  possible.  Once  you  have  finished  creating  the  unit 
please  press  the  next  button. 

Read  Text  (5  minutes).  -  In  this  task  you  should  read  out  aloud  the  text  presented 
at  the  bottom  of  the  screen.  I  will  record  your  responses  manually.  You  should  do 
this  as  quickly,  clearly  and  as  accurately  as  possible.  Once  you  have  finished 
reading  please  press  the  next  button. 
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First/Second  Session  Script 

•  Now  we  are  going  to  perform  the  first/ second  session  of  the  experiment  where  data 
will  be  collected.  In  this  session  I  will  ask  you  to  perform  the  tasks  you  just  observed 
under  a  no  motion  condition.  These  static  only  session  are  to  familiarise  you  with  all 
the  tasks.  There  will  be  three  sets  of  24  tasks,  and  a  5-10minute  break  in  between  each, 
where  you  will  be  asked  walk  outside  the  room. 

•  Show  NASA-TLX  -  After  you  complete  each  trial,  you  will  be  required  to  complete  the 
NASA  Task  Load  Index,  which  asks  you  to  rate  the  workload  of  each  trial  according  to 
your  experienced  mental  demand,  physical  demand,  temporal  demand,  performance, 
effort,  and  frustration  level.  Please  read  the  definitions  of  each  scale  on  this  definition 
sheet  (hand  sheet) 

•  Show  Questionnaire  - 1  will  also  ask  you  to  complete  this  questionnaire  after  each  trial. 
It  asks  you  to  rate  each  of  the  6  different  tasks  in  terms  of  difficulty  on  a  five  point 
scale. 

•  Give  Motion  Sickness  Susceptibility  Questionnaire  -  Please  complete  the  motion  sickness 
susceptibility  questionnaire. 

You  will  now  start  the  experimental  sessions  proper. 

•  For  Static  Trials  -In  this  trial  you  will  be  presented  with  24  tasks  presented  in  random 
order.  You  will  not  experience  any  motion.  You  should  complete  the  trials  as  quickly 

and  accurately  as  possible,  (on  completion) . Please  get  out  of  the  motion  simulator 

(sit  them  at  desk) 

•  Please  complete  this  Simulator  Sickness  Questionnaire  remembering  to  stand  for 
questions  12-14 

•  Now  please  complete  the  NASA  Task  Load  Index  by  marking  a  square  on  the 
scale  rating  the  workload  you  experienced  during  this  trial 


Thank  you  for  completing  this  session 
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Following  Session  Scripts 

Today  we  are  going  to  perform  the  next  session  of  the  experiment.  Just  as  during  the 
previous  session  I  will  ask  you  to  perform  24  randomly  assigned  tasks  during  three 
different  simulated  driving  conditions. 

Show  Simulator  Sickness  Questionnaire  -  Once  again  please  complete  this  Simulator 
Sickness  Questionnaire  to  assess  your  physical  state,  remembering  to  stand  when 
assessing  for  symptoms  12  to  14.  (Completes  questionnaire).  Also  please  remember 
that  you  should  let  me  know  immediately  if  you  begin  feeling  sick  during  your  time 
on  the  simulator.  At  the  end  of  each  trial  I  will  also  ask  you  complete  the  Simulator 
Sickness  Questionnaire  once  again. 

Remind  about  Forms  -  After  you  complete  each  trial,  I  will  also  ask  you  to  complete 

the  Simulator  Sickness  Questionnaire  once  again  as  well  as  the  NASA  Task  Load 
Index  and  the  task  ranking  questionnaire. 

Static  Trial  -For  this  trial  you  will  be  presented  with  24  tasks  presented  in  random 
order.  You  will  not  experience  any  motion.  You  should  complete  the  trials  as  quickly 

and  accurately  as  possible,  (on  completion) . Please  get  out  of  the  motion  simulator 

(sit  them  at  desk) 

•  Please  complete  this  Simulator  Sickness  Questionnaire  remembering  to  stand  for 
questions  12-14 

•  Now  please  complete  the  NASA  Task  Load  Index  by  marking  a  square  on  the 
scale  rating  the  workload  you  experienced  during  this  trial 

•  Finally  please  complete  this  questionnaire  by  ranking  each  of  the  6  different  tasks 
in  order  of  difficulty.  If  you  wish,  please  also  provide  any  other  comments  about 
the  difficulty  of  each  task. 

Mild  Motion  Trial  -  In  this  trial  you  will  once  again  be  presented  with  24  tasks 
presented  in  random  order.  This  time  however,  the  motion  chair  will  simulate  the 
vibrations  you  might  experience  while  driving  on  a  relatively  good  sealed  road.  You 
should  complete  the  trails  as  quickly  and  accurately  as  possible.  During  the  typing  trial 
you  should  try  and  correct  any  typing  mistakes  made,  but  if  this  proves  too  difficult 
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just  leave  that  word  as  it  is  and  try  and  type  the  next  word  in  the  list,  (on 
completion) . Please  get  out  of  the  motion  simulator  (sit  them  at  desk) 

•  Please  complete  this  Simulator  Sickness  Questionnaire  remembering  to  stand  for 
questions  12-14 

•  Now  please  complete  the  NASA  Task  Load  Index  by  marking  a  square  on  the 
scale  rating  the  workload  you  experienced  during  this  trial 

•  Finally  please  complete  this  questionnaire  by  ranking  each  of  the  6  different  tasks 
in  order  of  difficulty.  If  you  wish,  please  also  provide  any  other  comments  about 
the  difficulty  of  each  task. 

•  Rough  Motion  Trial  -for  this  trial  you  will  once  again  be  presented  with  24  tasks 

presented  in  random  order.  This  time  however,  the  motion  chair  will  simulate  the 
vibrations  you  might  experience  while  driving  on  a  relatively  rough  dirt  road.  You 
should  complete  the  trails  as  quickly  and  accurately  as  possible.  During  the  typing  trial 
you  should  try  and  correct  any  typing  mistakes  made'  but  if  this  proves  too  difficult 
just  leave  that  word  as  it  is  and  try  and  type  the  next  word  in  the  list,  (on 
completion) . Please  get  out  of  the  motion  simulator  (sit  them  at  desk) 

•  Please  complete  this  Simulator  Sickness  Questionnaire  remembering  to  stand  for 
questions  12-14 

•  Now  please  complete  the  NASA  Task  Load  Index  by  marking  a  square  on  the 
scale  rating  the  workload  you  experienced  during  this  trial 

•  Finally  please  complete  this  questionnaire  by  ranking  each  of  the  6  different  tasks 
in  order  of  difficulty.  If  you  wish,  please  also  provide  any  other  comments  about 
the  difficulty  of  each  task. 

Thank  you  for  completing  this  session 

•  (At  end  Might  want  to  offer  them  the  opportunity  to  come  back  and  play  the  racing 
simulation  in  the  chair  as  a  reward  one  lunch  time) 
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Appendix  G:  Create  Unit  Task 
Visual  Information  about  an  Enemy  Detection 


1.  The  type  of  enemy  unit  detected  is  represented  by  a  Military  Icon 


Infantry: 

Tank 

Artillery 

Helicopter 

Missile 

r 

JL 

i 

2.  The  Number  of  detected  enemy  is  indicated  by  the  symbol 
displayed  on  top  of  the  red  diamond 


One  enemy:  • 

Two  enemies:  •• 

Three  enemies:  ••• 


that  is  a  single  black  dot 
that  is  two  black  dots  in  a  row 
that  is  three  black  dots  in  a  row 


3.  The  enemy  will  appear  on  the  map  display  as  follows: 


Read  out  aloud  the  TYPE  SIZE  and  LOCATION  of  the  new  enemy  detection 


Next  Task 
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Appendix  H:  BMS  Task  Difficulty  vs  Sessions  - 

Learning  Effects 


The  BMS  Task  Difficulty  Questionnaires,  may  also  give  insight  into  learning  effects.  The 
following  are  graphs  of  how  difficult  participants  rated  the  task  over  the  six  sessions. 


Writing  Task  Difficulty  vs  Session 


Condition 

“—static 

motion2 

motion4 


Reading  Task  Difficulty  vs  Session 


Condition 

^“static 
—  motion2 
motion4 
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Pan-Zoom  Task  Difficulty  vs  Session 


Condition 

—  static 

—  motion2 
motion4 


Create  Line  Task  Difficulty  vs  Session 


Condition 

—  static 

—  motion2 
motion4 
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Read  Unit  Task  Difficulty  vs  Session 


Condition 

^“static 

- motion2 

motion4 


Create  Unit  Task  Difficulty  vs  Session 


Condition 

-static 
—  motion2 
motion4 
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